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Purpose 


^  First,  present  our  approach,  methodology 
and  findings  that  we  used  and 
encountered  during  our  V&V  of  the  USMC 
WRMR  Model. 


Second,  present  and  offer  some  insights, 
traps,  and  lessons  learned  during  our  V&V 
effort. 
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WRMR  V&V  Objectives 

#To  improve  WRMR  model  and 
procedures. 

#To  comply  with  CMC  order  directing  that 
DC,  CD&I  coordinate  VV&A  following  the 
Naval  Audit  Service  report  in  June  2006. 

^  Complete  V&V  to  support  model 
accreditation  for  POM-1 0  analysis. 
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V6&V  Related  Activities 


‘ER 


MRP  DoDI  3000.4 


#  Align  the  Marine  Corps 
munitions  requirements 
process  (MCMRP)  with 
DoD  instructions  and 
guidance 

#  Support  the  MCMRP 
and  the  POM-1 0 
submission  through 
WRMR  model 
development 


#  V&V  of  the  WRMR 
model 
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Munitions  Requirements  Process 

DoDI  3000.4 


£5 


»‘ER 


“The  DoD  MRP  Implementation 


WRMR  Model  Process 


etc... 

JMEMS 


SME 


OPLANS 


Threat  Report 
TPFDD 

Phase-Threat 
Distribution 


Input 


V 


Model 


Compute  Combat 
Planning  Factors 


Compute 

MCO  Consumption 

(for  each  scenario) 

1 — ^ 


& 


Compute  MCO 
Combat  Loads 

(for  each  scenario) 

computational  inginl 


Output 


WRMR  is  deterministic  model  that  calculates  the  unconstrained 
munitions  requirements  for  the  USMC 
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Employed  V&V  Techniques 


#  Audit  -  an  informal  V&V  technique  used  to  assess  how  adequately  a 
model  or  simulation  is  used  with  respect  to  established  plans,  policies, 
procedures,  standards,  and  guidelines.  Required  an  in-depth  document 
review  and  comparison  of  the  model  to  DoDI  3000.4. 

#  Functional  testing  -  a  V&V  technique  used  to  assess  the  accuracy  of 
model’s  input-output  transformation. 

#  Sensitivity  analysis  -  a  V&V  technique  where  selected  model  inputs 
are  systematically  changed  over  a  range  of  interest  while  observing  the 
effect  upon  model  outputs. 

Face  validation  -  the  review  of  model  output  by  the  model  development 
team,  users  and  subject  matter  experts  for  reasonableness. 

Code  Walkthrough  -  an  examination  of  the  model  source  code  to 
detect  and  document  errors.  Conducted  with  the  model  developer. 

#  Regression  Testing  -  a  testing  technique  used  to  ensure  that 
corrections  and  modifications  to  the  model  do  not  create  other  errors  or 
adverse  side  effects.  Requires  that  the  modified  model  be  tested  with 
data  sets  used  by  the  previous  version  of  the  model.  Completed  by  the 
model  developer. 
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Source:  Defense  Modeling  and  Simulation  Office,  V&V  Techniques, 

Recommended  Practices  Guide,  Reference  Pianning  Document,  dtd  8/15/01 


Combat  Requirement  Audit 


WRMR  Model 
Implementation 
MCO  req:  For  each  WD 
and  SD,  compute 
combat  consumption 
plus  one  combat  load. 

Select  the  WD  and 
largest  of  two  SD  MCO 
expenditures  for  each 
DODIC. 


Combat  Requirement 
(CR) 

Current 

Operations/Forward 
Presence  Requirement 
(CO/FPR) 


War  Reserve 
Munitions 
Requirement 


Strategic  Readiness  [j 
Requirement  (SRR) 


WRMR  Model  Implementation 
SSC  req:  Calculated  using  combat 
planning  factors,  30  day  assault 
plus  1  day  combat  load  for  each  of  3 
MEB  sized  forces. 
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Combat  Requirement  Audit 

POM-08  Process 


155  HE 

81mm  HE 

Requirement 

Source 

MCO  (SD) 

9,725 

104,362 

DoDI  3000.4 

MCO  (WD) 

78,720 

168,678 

DoDI  3000.4 

1  MPS  pluK  1  r'^nih^f  tnar^nr  a  MEB  I 

48,510 

38,940 

AO  memo  dtd?K7I^^^Q6 

V  B  (2) 

48,510 

38,940 

AO  memo  dtd  2  May  0^ 

48.510 

38,940 

AO  meme--diTrfM^  06 

Combat  Load  -  based  on  number  of  weapons  in  the  force 

MCO  (SD) 

1,345 

8,147 

DoDI  3000.4 

MCO  (WD) 

13,433 

12,001 

DoDI  3000.4 

Total 

248,753 

410,008 

Recommend  policy  review  of  combat  requirement 


Model  Component  Verification  Example 


Functional  Testing  of  MCO  Target  Allocations  and  Kills  example 

•  Procedure 

-  Examined  Inputs/outputs  to  verify  MCO  aiiocations/kiiis/expenditures  for  a  particuiar 
weapon/munitions 

•  Target  Data 

-  Totai  Targets  =  1038  (DiA  TR) 

-  USMC  Aiiocation  =  6.2%  =  64.4  (COCOM  PTD) 

•  Shooter  Input  Data 

-  Rounds  per  kiii  =  1.11 

-  Shooter  Aiiocation  =  24.2%  =  15.6  Targets  to  Kiii 

-  Expected  number  of  rounds  expended  =  17.3 

©  Shooter/Model  Output  Data 

©  Targets  kiiied  =  15.6 
©  Rounds  expended  =  17.3 
©  Totai  Targets  kiiied  (aii  shooters)  =  64.4 
©  Verification 

©  Expected  Targets  to  Kiii  (Shooter)=  Targets  kiiied  (Shooter)  -  VERIFIED 
©  Expected  rounds  expended  (Shooter)  =  Rounds/kill  *  Targets  to  kill  (Shooter)  -  VERIFIED 
©  USMC  Target  Allocation  =  Total  Targets  killed  (All  shooter  types)  -  VERIFIED 


19  of  25  model  components  examined  and  verified. 
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Face  Validation  Example 

Total  Munitions  Breakout 


‘ER 


226,589,365  319,966  655,959 


120,848 


100% 


80% 


60% 


40% 


20% 


394,071 


49% 


28% 


37% 


11% 


23% 


u  /o  ^ 

A059  (5.56MM) 

C871  (81  MM  Ilium) 

C868  (81  MM  HE) 

D505  (155MM  Ilium) 

D529  (155MM  HE) 

□  Training/Testing 

140,160,325 

25,331 

170,035 

11,677 

89,449 

■  GWOT 

166,479 

0 

0 

0 

0 

□  Homeland  Def 

603,795 

2,601 

4,023 

0 

0 

□  MCPP-N 

16,413,580 

10,168 

33,632 

7,937 

29,106 

□  UDP 

426,000 

96 

708 

120 

1,368 

■  ACMs 

877,356 

2,346 

8,328 

0 

0 

□  LFORMs 

5,446,590 

8,595 

29,225 

6,715 

25,395 

□  SSC 

22,621,581 

35,641 

116,820 

39,684 

145,531 

□  WD 

05,549,43^ 

d  157,43^ 

CJ  80,670 

d44,4760 

CL92,1530 

□  SD 

14,324,222 

77,758 

112,509 

10,239 

9,127 

100% 


80% 


60% 


Face  Validation  Example 

Ammunition  Breakout  (WD) 

25,549,437  157,430  180,679  44,476  92,153 


‘ER 


Non¬ 

target 

oriented 


Yo 


20% 


1  U7o  J 

A059  (5.56  MM) 

C871  (81MM  liium) 

0868  (81  MM  HE) 

D505  (155MM  liium) 

D529  (155MM  HE) 

□  Losses 

0 

2 

6 

0 

0 

■  C2 

0 

1,164 

0 

8,112 

0 

□  Registration 

0 

Rin9 

^^96,598^ 

0 

262 

□  Rear  Area  Sec 

Cj335,660 ^  ^ 

1  ^ 

0 

0 

0 

□  liium 

0 

C^146,069_^ 

0 

33,924 

Ok  0 

■  Zero 

2,439,860 

0 

0 

0 

\  ° 

□  OpCheck 

670,903 

0 

0 

0 

\  0 

□  Reoiace 

1,058,234 

8,834 

9,547 

1,261 

^  2,293 

■  Tgt  Oriented 

12,389^^^  Seems  small 

62,527 

0 

^76,165"^ 

□  Combat  Load 

16,932,384 

1,361 

12,001 

1,179 

13,433 

Recommendations 


Valid  for 
requirements 
determination  & 
CPF  use 


Valid  for 
requirements 
determination 


Not  valid  for 
requirements 
determination 


Lessons  Learned 


#  Legacy  model  V&V  is  challenging  -  following  website  proved  to  be 
very  helpful: 

#  Our  review  of  the  WRMR  conceptual  model  was  important; 
however,  we  learned  a  great  deal  more  about  the  model  by 
running  it. 

#  The  WRMR  model  depends  upon  a  large  amount  of  data  -  not  all 
owned  by  the  USMC. 

#  Determining  your  basis  of  comparison  is  not  always  straight 
forward!  In  defining  our  basis  of  comparison  for  the  WRMR 
model,  we  looked  at  the  other  services  models. 

#  Documentation,  documentation,  documentation! 

#  Establish  contact,  build  a  rapport  with  the  developer  and  user  - 
smooth  relations  promotes  better  overall  support. 

Walking  through  the  WRMR  code  with  the  developer  was 
extremely  helpful. 

#  The  WRMR  V&V  benefited  from  the  interdisciplinary  makeup  of 
the  team  -  computer  science,  operations  research,  subject  matter 
expertise,  and  Marine  Corps  expertise. 
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Backup 
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WRMR  V&V  Timeline 


USMC  Munitions  Requirements  Process 


Document  Review 


#  In-depth  review  of  appropriate  documents  was 
accomplished  to  gain  an  understanding  of  the 
munitions  requirements  process  and  to  confirm  M&S 
requirements  in  order  to  perform  conceptual  model 
validation. 

^  DoDI  3000.4,  MCO  8000.7,  MCO  8000.8  (draft) 

^  POM  08  TMR 

#  CMC  Itr  19  Jun  06  (Response  to  Naval  Audit) 

^  MCMRP  Brief  dtd  Nov  06 

^  DoD  MRP  POM-1 0  Conference  Brief,  dtd  26  Sep  06 

#  Modei  documentation  was  reviewed  to  gain 
understanding  of  modei  processes. 

©  Computational  model  well  documented 

©  Model  input  and  output  documentation  updated  to  inciude 
fiie  descriptions. 


Target-Oriented  Expenditures 


#  By  DoD  INST  3000.4,  total  targets  to  be  defeated  are 
obtained  from  the  DIA  Threat  Report  (TR)  and  COCOM 
Phased  Threat  Distribution  (PTD): 


Tgts  to  be  Defeated  {by  phase)  =  Tot  Tgts  (77?)  *  Service  Allocation  {PTD) 


#  The  WRMR  computational  engine  determines  daily 
expenditures  based  on  the  number  of  targets  killed 
over  the  course  of  a  particular  phase  in  order  to 
support  the  calculation  of  combat  planning  factors 
(CPFs). 


Source:  Class  V(W)  Requirements  Methodology  (POM-08)  no  date 
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Non-target  Expenditures 


#  Registration 

#  Zeroing 

#  Operationai  check 
iiiumination 

#  Mine 

#  Screening 


#  Expiosive  ordnance 

disposai 

Demoiition 

#  Command  and 
controi 

#  Rear-area  security 

#  Seif-defense 

#  Obscuration 


Non-target  expenditures  account  for  munitions 
requirements  not  directiy  reiated  to  target  destruction. 


Non-target  Expenditures 

Illumination  Example 


Situation:  MAGTF  consisting  of  9  Inf  Co’s  employed  to  date  in  the 
defense,  2/3  of  which  are  engaged. 

Problem:  Calculate  the  daily  requirement  for  illumination  in  this  situation. 


Equation: 


Daily  Expenditure  =  (Num  Inf  Co)*(Frac  Engaged) 


*  (Required  ilium  (min)) 
(Ilium  provided  by  rnd  (min)) 


Variables: 

Num  Inf  Co  =  9 

Frac  Engaged  =  2/3 

Required  Ilium  (min)  =  32  (min) 

Ilium  provided  by  rnd  (min)  =  1  (min)  (DODIC  C871) 


Answer: 

C871  =  9  *  %  *  (32/1)  =  192  Rnds  per  day 


Source;  Class  V(W)  Requirements  Methodology  (POM-08) 


Combat  Planning  Factors 


Determine  average  daily  expenditure  per  shooter 
type  for  all  scenarios. 

#  Sort  average  daily  expenditure  from  high  to  low. 

#  Identify  and  group  high-intensity  days  and  low- 
intensity  days. 

#  Calculate  CPFs 

#  Assault  rate  =  Average  expenditure  rate  of  high-intensity 
days 

#  Sustain  rate  =  Average  expenditure  rate  of  low-intensity 
days 


Combat  planning  factors  are  a  key  element  of 
munitions  requirements  determination. 


23 


Current  Operations/Forward  Presence  (CO/FP. 


‘ER 


ACM  -  air  contingency  MAGTF 

LFORM  -  landing  force  operational  reserve  materiel 
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strategic  Readiness  Requirement 


‘ER 


WRMR  Model 
Implementation: 
Homeland  defense 
is  30  days  sustain 

plus  1  combat  load 
for  3  infantry 
battalions. 


Current 

Operations/Forward 
Presence  Requirement 
(CO/FPR) 


Strategic 
Readiness 
Requirement  (SRR) 


WRMR  Model  Implementation: 
GWOT  is  3  days  assauit  plus  15 
days  sustain  plus  1  combat  load 
for  3  AT  teams. 


WRMR  Model 
Implementation:  UDP  is 

1  combat  load  for  1 
infantry  battalion,  1  arty 
battalion,  1  AAV  company 
and  1  LAR  company. 


WRMR  Modei 
Impiementation:  MCPP- 
N  is  30  day  assauit  rate 
plus  1  combat  load  for 
force  identified  in  MCBul 
3502. 


AT  -  anti-terrorist 

LFORM  -  landing  force  operational  reserve  materiel 

UDP  -  unit  deployment  program 

MCPP-N  -  Marine  Corps  pre-positioned  -  Norway 
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Model  Component  Verification  Summary 


Model  Components 

Verified 

Method 

Determine  Expenditures  per  Target 

Yes 

Functionai  Test 

Push  Targets  to  Underutiiized  Shooters 

Yes 

Functionai  Test 

Scaie  Target  Aiiocation 

Yes 

Code  Waikthrough 

Assess  USMC  Losses 

Yes 

Functionai  Test,  Code 
Waikthrough 

Determine  Combat  Losses  of  Munitions 

Yes 

Functionai  Test 

Determined  Repiacement  and/or  Repairs 

Yes 

Functionai  Test 

Determine  Registration  Expenditures 

Yes 

Functionai  Test 

Determine  Zeroing  Expenditures 

Yes 

Functionai  Test 

Determine  Operationai  Check  Expenditures 

Yes 

Functionai  Test 

Determine  iiiumination  Expenditures 

Yes 

Functionai  Test 

Determine  Obscuration  Expenditures 

Not  tested 

Doc.  &  input  Data  Review 

Determine  Screening  Expenditures 

Not  tested 

Doc.  &  input  Data  Review 

Determine  Demoiition  Expenditures 

Not  tested 

Doc  &  input  Data  Review 

Determine  Expiosives  Ordnance  Disposai  (EOD) 
Expenditures 

Not  tested 

Doc  &  input  Data  Review 

r 

Model  Component  Verification  Summary  (Cont) 


Model  Components 

Verified 

Method 

Determine  Mine  Expenditures 

Not  tested 

Doc.  &  Input  Data  Review 

Determine  Command  &  Control  Expenditures 

Yes 

Functional  Test 

Determine  Rear  Area  Security  Expenditures 

Yes 

Functional  Test 

Determine  Self  Defense  Expenditures 

Yes 

Functional  Test 

Determine  Ancillary  Expenditures 

Yes 

Functional  Test 

Compute  Win  Decisively  (WD)  and  Swiftly  Defeat  the 

Effort  (SDTE)  Requirements 

Yes 

Functional  Test,  Audit,  Face 
Validation,  Sensitivity  Analysis 

Compute  Combat  Planning  Factors  (CPF) 

Yes 

Functional  Test,  Audit,  Face 
Validation,  Code  walkthrough 

Compute  Small-Scale  Contingency  Requirements 
(SSCR) 

Yes 

Functional  Test,  Audit,  Face 
Validation,  Sensitivity  Analysis 

Compute  Current  Operations/  Forward  Presence 

Requirements  (CO/FPR) 

Yes 

Functional  Test,  Audit,  Face 
Validation,  Sensitivity  Analysis 

Compute  Strategic  Readiness  Requirement  (SRR) 

Yes 

Functional  Test,  Audit,  Face 
Validation,  Sensitivity  Analysis 
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Model  Component  Validation  Summary  (i  of 


Model  Components 

Valid  ? 

Discussion 

Determine  Expenditures  per  Target 

V 

Uses  JMEMS  methodology; 

Push  Targets  to  Underutilized  Shooters 

V 

Allows  allocation  of  targets  from 
over-taxed  shooters  to  under-utilized 
shooters 

Scale  Target  Allocation 

V 

Scales  down  target  allocations  so 
that  busiest  shooter  types  to  do  not 
exceed  daily  limit  and  weapon-tgt 
preferences  remain  consistent 

Assess  USMC  Losses 

NV 

Review  ’04  SME  data;  review 
proportionality  assumption 

Determine  Combat  Losses  of  Munitions 

NV 

Method  may  double  count  ammo  lost 
When  weapon  system  is  lost 

Determined  Replacement  and/or  Repairs 

V 

Determines  fraction  of  killed  tgts 
expected  to  return 

Determine  Registration  Expenditures 

NV 

Review  data  and  methodology 

Preliminary  Assessment 

V:  Valid,  NV:  Not  valid 


Model  component  not  valid  for  POM-1 0 
Time  permitting,  review  for  POM-1 0 
Sufficient  for  POM-1 0 
Corrected  during  V&V 
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Model  Component  Validation  Summary  (2  of 


Model  Components 

Valid  ? 

Discussion 

Determine  Zeroing  Expenditures 

V 

Review  ’04  SME  data  only 

Determine  Operational  Check  Expenditures 

V 

Review  ’04  SME  data  only 

Determine  Illumination  Expenditures 

NV 

Face  validation  indicates  high 
expenditures;  methodology 
may  be  too  complex,  review  with 

SME,  align  model  phases 
with  OPLANS 

Determine  Obscuration  Expenditures 

NV 

Face  validation 

Determine  Screening  Expenditures 

V 

Review  ’04  SME  data  only 

Determine  Demolition  Expenditures 

NV 

Align  OPLANS/model 

Determine  EOD  Expenditures 

NV 

Align  OPLANS/model 

Determine  Mine  Expenditures 

NV 

Align  OPLANS/model 

Determine  Command  &  Control  Expenditures 

NV 

Align  OPLANS/model 

Determine  Rear  Area  Security  Expenditures 

V 

Review  ’04  SME  data  only 

Preliminary  Assessment 

V:  Valid,  NV:  Not  valid 


Model  component  not  valid  for  POM-1 0 
Time  permitting,  review  for  POM-1 0 
Sufficient  for  POM-1 0 
Corrected  during  V&V 


Model  Component  Validation  Summary  (3  of 


Model  Components 

Valid  ? 

Discussion 

Determine  Self  Defense  Expenditures 

V 

Review  ’04  SME  data  only 

Determine  Ancillary  Expenditures 

V 

Review  ’04  SME  data  only 

Compute  Win  Decisively  (WD)  and  Swiftly  Defeat  the 
Effort  (SDTE)  Requirements 

V 

Align  OPLANS/model 

Compute  Combat  Planning  Factors  (CPF) 

V 

Review  non-target  oriented 
munitions  results 

Compute  Small-Scale  Contingency  Requirements 
(SSCR) 

NV 

Requirements  under  revision 

Compute  Current  Operations/  Forward  Presence 
Requirements  (CO/FPR) 

V 

Requirements  under  revision 

Compute  Strategic  Readiness  Requirement  (SRR) 

NV 

Must  comply  with  DoDI  3000.4 
Requirements  under  revision 

Preliminary  Assessment 

V:  Valid,  NV:  Not  valid 


Model  component  not  valid  for  POM-1 0 
Time  permitting,  review  for  POM-1 0 
Sufficient  for  POM-1 0 
Corrected  during  V&V 
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